User information extraction method and user information extraction apparatus

ABSTRACT

The present application discloses a user information extraction method and a user information extraction apparatus. The method comprises: acquiring an image comprising at least one digital watermark; acquiring user related information corresponding to a user and comprised in the at least one digital watermark in the image; and projecting the user related information to a fundus of the user. In the present application, user related information used in a certain scenario is extracted in the corresponding scenario and projected onto a fundus of a user, and therefore, the user does not need to memorize the user related information, thereby avoiding inconvenience caused when the user forgets the user related information.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a national stage application of InternationalApplication No. PCT/CN2014/071143, filed on Jan. 22, 2014, which claimspriority to and the benefit of Chinese Patent Application No.201310572121.9 filed with the State Intellectual Property Office of P.R.China on Nov. 15, 2013, and entitled “USER INFORMATION ACQUISITIONMETHOD AND USER INFORMATION ACQUISITION APPARATUS”. The contents of bothof the above-referenced applications are herein incorporated byreference in their entirety.

TECHNICAL FIELD

The present application relates to the field of digital authenticationtechnologies, and in particular, to a user information extraction methodand apparatus.

BACKGROUND

In order to save energy and avoid misoperations, mobile or wearabledevices are generally set with a screen locking function, and a lockscreen may be unlocked with or without a password. When a screen with apassword is to be unlocked, a user needs to memorize some specialpasswords, patterns, actions, or the like. Although security can beensured, a case in which the user forgets those special passwords,patterns, actions, or the like occurs easily, which brings inconvenienceto the user. Certainly, in addition to the foregoing scenario ofunlocking the screen, the foregoing problem also exists in otherscenarios in which passwords need to be input for further operations.

By using digital watermark technologies, some identifier information(that is, the digital watermark) may be directly embedded in a digitalcarrier without affecting using of the original carrier, and theidentifier information is not detected and modified easily. The digitalwatermark technologies are applied to many aspects, such as copyrightprotection, anti-counterfeiting, authentication, and information hiding.If the digital watermark technologies can be used to securely andcovertly help the user input a password, or the like, to obtaincorresponding authorization, the foregoing problem that authenticationcannot be performed because the user forgets the password may be solved,thereby improving user experience.

SUMMARY

An objective of the present application is to provide a user informationextraction solution, so as to provide a user, without breachingconfidentiality, user related information used in a correspondingscenario, thereby avoiding inconvenience caused by the user forgettingthe corresponding user related information.

To achieve the foregoing objective, according to a first aspect, thepresent application provides a user information extraction method,comprising:

acquiring an image comprising at least one digital watermark;

acquiring user related information corresponding to a user and comprisedin the at least one digital watermark in the image; and

projecting the user related information to a fundus of the user.

According to a second aspect, the present application provides a userinformation extraction apparatus, comprising:

an image acquisition module, configured to acquire an image comprisingat least one digital watermark;

an information acquisition module, configured to acquire user relatedinformation corresponding to a user and comprised in the at least onedigital watermark in the image; and

a projection module, configured to project the user related informationto a fundus of the user.

In at least one technical solution of embodiments of the presentapplication, user related information used in a certain scenario isextracted in the corresponding scenario and projected onto a fundus of auser, and therefore, the user does not need to memorize the user relatedinformation, thereby avoiding inconvenience caused by the userforgetting the user related information.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a flowchart of steps of a user information extraction methodaccording to an embodiment of the present application;

FIG. 2a to FIG. 2c are schematic application diagrams of a userinformation extraction method according to an embodiment of the presentapplication;

FIG. 3a and FIG. 3b are schematic application diagrams of another userinformation extraction method according to an embodiment of the presentapplication;

FIG. 4a and FIG. 4b are schematic diagrams of a light spot pattern usedin a user information extraction method according to an embodiment ofthe present application and an image obtained at a fundus and comprisingthe light spot pattern;

FIG. 5 is a schematic block diagram of a structure of a user informationextraction apparatus according to an embodiment of the presentapplication;

FIG. 6a and FIG. 6b are schematic block diagrams of structures ofanother two user information extraction apparatuses according to anembodiment of the present application;

FIG. 7a is a schematic block diagram of a structure of a locationdetection module used in a user information extraction apparatusaccording to an embodiment of the present application;

FIG. 7b is a schematic block diagram of a structure of a locationdetection module used in another user information extraction apparatusaccording to an embodiment of the present application;

FIG. 7c and FIG. 7d are corresponding optical path diagrams when alocation detection module used in a user information extractionapparatus according to an embodiment of the present application performslocation detection;

FIG. 8 is a schematic diagram of applying a user information extractionapparatus to glasses according to an embodiment of the presentapplication;

FIG. 9 is a schematic diagram of applying another user informationextraction apparatus to glasses according to an embodiment of thepresent application;

FIG. 10 is a schematic diagram of applying still another userinformation extraction apparatus to glasses according to an embodimentof the present application;

FIG. 11 is a schematic structural diagram of yet another userinformation extraction apparatus according to an embodiment of thepresent application;

FIG. 12 is a flowchart of steps of a user information embedding methodaccording to an embodiment of the present application; and

FIG. 13 is a schematic block diagram of a structure of a userinformation embedding apparatus according to an embodiment of thepresent application.

DETAILED DESCRIPTION

The methods and apparatus according to the present application aredescribed in detail with reference to accompanying drawings andembodiments as follows:

In everyday life, a user often needs to use various user relatedinformation, for example, a user password or a special gesture thatneeds to be input by the user on a lock screen interface of electronicdevices, a user password that needs to be used by the user when loggingin to some web sites or an account of an application, passwordinformation that needs to be used in some access control devices, orother various user authentication information. The user needs tomemorize these various user related information, and input correspondinguser related information in a corresponding scenario to perform afurther operation; otherwise, great inconvenience may be caused to theuser. Therefore, in the present application, as shown in FIG. 1, anembodiment of the present application provides a user informationextraction method, comprising:

S120: Acquire an image comprising at least one digital watermark.

S140: Acquire user related information corresponding to a user andcomprised in the at least one digital watermark in the image.

S160: Project the user related information to a fundus of the user.

In an embodiment of the present application, user related informationrelated to a user is acquired from a digital watermark of an image andprojected onto a fundus of the user, to cause the user to confidentiallyacquire, without special memorization, the corresponding user relatedinformation in a scenario in which the corresponding user relatedinformation needs to be used, thereby facilitating use of the user, andimproving user experience.

In the following, an embodiment of the present application furtherdescribes the steps by using the following implementation manner:

S120: Acquire an image comprising at least one digital watermark.

There are various manners for acquiring the image in the embodiments ofthe present application, for example:

1. Acquire the image by means of shooting.

In an embodiment of the present application, a smart glasses device maybe used to shoot an object seen by the user. For example, when the usersees the image, the smart glasses device shoots the image.

2. Acquire the image by means of receiving from an external device.

In a possible implementation manner of an embodiment of the presentapplication, the image may further be acquired by using other devices,or the image is acquired through interaction with a device displayingthe image.

S140: Acquire user related information corresponding to a user andcomprised in the at least one digital watermark in the image.

In an embodiment of the present application, there are various methodsfor acquiring the user related information, for example, one or more ofthe following:

1. Extract the user related information from the image.

In this implementation manner, for example, the digital watermark in theimage may be analyzed by using a personal private key and a public orprivate watermark extraction method, to extract the user relatedinformation.

2. Send the image to an external device, and receive the user relatedinformation in the image from the external device.

In this implementation manner, the image may be sent to an externaldevice, for example, sent to a cloud server or a third party authorityto extract the user related information in the digital watermark in theimage by using the cloud server or the third party authority.

In a possible implementation manner of an embodiment of the presentapplication, the image corresponds to a graphical user interfacedisplayed by a device. As described above, the graphical user interfacemay be, for example, a lock screen of the device (as shown in FIG. 2aand FIG. 3a ), or may further be an input interface of userauthentication information, such as a password for an application, a website, or the like.

In a possible implementation manner, the graphical user interfacecomprises an input interface of the user related information, such astrack keys 230 and number keys 330 shown in FIG. 2a and FIG. 3a , tocause the user to input corresponding user related information.

In a possible implementation manner of an embodiment of the presentapplication, the user related information comprises user authenticationinformation corresponding to the image.

The user authentication information herein may be, for example, the userpassword or a special posture (such as a hand gesture, or an overallbody posture) described above. That the user authentication information“corresponds to the image” may be, for example, that a lock screen of anelectronic device (such as a mobile phone, a tablet computer, a notebookcomputer, or a desktop computer) is displayed in the image, wherein thelock screen comprises a user authentication information input promptinterface to prompt the user to input corresponding user authenticationinformation, and the user related information obtained by the useraccording to the image is the to-be-input user authenticationinformation. Alternatively, for example, an image (the image may be, forexample, a static image generated by printing or the like, or may be anelectronic image displayed by using an electronic device) comprising awatermark exists near an access controller to which a password needs tobe input. The user may acquire password information of the accesscontroller from the image by using the foregoing method.

Certainly, in another possible implementation manner of the embodimentsof the present application, the user related information may further beother information, for example, the image is a user environmentinterface displayed by an electronic device (such as a computer) sharedby multiple people. When one of the users uses the method according toan embodiment of the present application, the user can acquire, from theimage, input information to access a corresponding application on theenvironment interface associated with the user, which facilitates use ofthe user.

In a possible implementation manner of the embodiments of the presentapplication, before step S140, user authentication may furtherperformed, wherein an identity of the user is confirmed, to cause theuser related information corresponding to the user to be obtained instep S140. For example, the user performs the functions of the steps inthe embodiments of the present application by using a pair of smartglasses on which identity authentication is successfully performed.

S160: Project the user related information to a fundus of the user.

In an embodiment of the present application, in order to cause the userto obtain the user related information in a confidential manner, theuser related information is projected onto the fundus of the user tocause the user to obtain the corresponding user related information.

In a possible implementation manner, the projection may be that the userrelated information is directly projected onto the fundus of an eye ofthe user by using a projection module.

In another possible implementation manner, the projection may also bethat the user related information is displayed at a location that onlythe user can see (for example, a display surface of a pair of smartglasses), and the user related information is projected onto the fundusof the user through the display surface.

In the first manner, the user related information does not need to passthrough an intermediate display but directly reaches the fundus of theuser, and therefore, the first manner has higher privacy. The followingfurther describes this implementation manner. The step of projecting theuser related information onto a fundus of the user comprises:

projecting the user related information; and

adjusting at least one projection imaging parameter of an optical pathbetween a projection location and an eye of the user until the image ofthe user related information formed at the fundus of the user satisfiesat least one defined first clarity criterion.

Herein, the clarity criterion may be set according to a claritymeasurement parameter commonly used by a person skilled in the art, suchas effective resolution of an image.

In a possible implementation manner of the embodiment of the presentapplication, the adjusting at least one projection imaging parameter ofan optical path between the projection location and an eye of the usercomprises:

adjusting at least one imaging parameter of at least one optical deviceon the optical path between the projection location and the eye of theuser, and/or a location of the at least one optical device on theoptical path.

The imaging parameters herein may include a focal length, an opticalaxis direction, and the like of optical devices. The adjustment causesthe user related information to be properly projected onto the fundus ofthe user, for example, by adjusting a focal length of the at least oneoptical device, an image of the user related information can be clearlyformed on the fundus of the user. “Clearly” herein refers to satisfyingthe at least one defined first clarity criterion. Alternatively, in thefollowing description, when three-dimensional display is required, inaddition to directly generating an image of the left eye and an image ofthe right eye that are with parallax, the user related informationcorresponding to two eyes of the user may be the same, and an effect ofthree-dimensional display of the user related information may beachieved by separately projecting the user related information to thetwo eyes with certain deviation. In such cases, for example, an opticalaxis parameter of the optical device may be adjusted.

When the user sees the user related information, a direction of a sightline of an eye may change, and it is required that the user relatedinformation be well projected onto the fundus of the user at the timewhen the directions of the sight line of the eye are different.Therefore, in a possible implementation manner of an embodiment of thepresent application, step S160 further comprises:

transferring, corresponding to locations of a pupil when optical axisdirections of the eye are different, the user related information to thefundus of the user.

In a possible implementation manner of an embodiment of the presentapplication, a curved optical device, such as a curved beam splitter maybe required to implement the function of the foregoing step. However,to-be-displayed content is generally deformed after passing through thecurved optional device. Therefore, in a possible implementation mannerof an embodiment of the present application, step S160 furthercomprises:

performing, on the user related information, counter-deformationprocessing corresponding to locations of a pupil at the time when theoptical axis directions of the eye are different, to cause the fundus toreceive the user related information that needs to be presented.

For example, the projected user related information is preprocessed tocause the projected user related information to have acounter-deformation effect that is opposite to the deformation, and thenafter the user related information passes through the foregoing curvedoptical device, the counter-deformation effect is offset by adeformation effect of the curved optical device, and therefore, the userrelated information received at the fundus of the user is presented witha required effect.

In a possible implementation manner, user related information projectedonto the eye of the user does not need to be aligned with the image. Forexample, when the user needs to input a group of password in sequence,for example, “1234”, the group of password only needs to be projectedonto the fundus of the user to be seen by the user. However, in somecases, for example, when the user related information is that a specificaction is performed at a specific location, for example, a specifictrack is to be drawn at a specific location described in FIG. 2b , theuser related information needs to be displayed and aligned with theimage. Therefore, in a possible implementation manner of an embodimentof the present application, step S160 comprises:

aligning, at the fundus of the user, the projected user relatedinformation with an image seen by the user.

As shown in FIG. 2a and FIG. 2b , in this implementation manner, theuser sees an image 210 shown in FIG. 2a , wherein the image is, forexample, a lock screen interface of a mobile phone device, and the useracquires corresponding user related information 220 according to theimage 210. In step S160, after being aligned with the image 210 (theuser related information is presented at a corresponding location in theimage), the user related information 220 is projected onto the fundus ofthe user, to cause the user to see the picture shown in FIG. 2b ,wherein a track shown by a dashed line is the user related information220 projected onto the fundus of the user. After the user inputscorresponding track on the lock screen according to the user relatedinformation (to obtain an image shown in FIG. 2c ), the mobile phone isunlocked.

FIG. 3a and FIG. 3b are another implementation manner. The user firstsees an image 310 shown in FIG. 3a , wherein the image is, for example,a lock screen interface of a mobile phone device, and corresponding userrelated information 320 is acquired according to the image 310, forexample, a dashed box shown in FIG. 3b , wherein corresponding numberkeys 330 in the round box are numbers that are to be input subsequently.

To implement the foregoing alignment function, in a possibleimplementation manner, the method further comprises:

detecting a location, of a gaze point of the user, relative to the user,wherein

the aligning, at the fundus of the user, the projected user relatedinformation with an image seen by the user comprises: aligning, at thefundus of the user according to the location of the gaze point of theuser, relative to the user, the projected user related information withthe image seen by the user.

Herein, because the user is looking at the image at this moment, forexample, a lock screen interface of a mobile phone of the user, thelocation corresponding to the gaze point of the user is the location atwhich the image is located.

In this implementation manner, there are various manners for detectingthe location of the gaze point of the user, which for example, compriseone or more of the following:

1) Use a pupil direction detector to detect an optical axis direction ofan eye, and use a depth sensor (such as an infrared distance meter) toobtain a depth of an eye-gazed scenario, and obtain a location of a gazepoint of a sight line of the eye. The technology belongs to the priorart, and is not described again in this implementation manner.

2) Separately detect optical axis directions of two eyes, obtaindirections of sight lines of the two eyes of the user according to theoptical axis directions of the two eyes, and obtain the location of thegaze point of the sight lines of the eyes by using an intersection pointof the directions of the sight lines of the two eyes. The technologyalso belongs to the prior art, and is not described herein again.

3) Obtain the location of the gaze point of the sight line of the eyeaccording to optical parameters of an optical path between an imageacquisition location and the eye and the optical parameters of the eyeat the time when a fundus image that satisfies at least one definedsecond clarity criterion is presented on an imaging surface of an eye isacquired. The embodiments of the present application provide a specificprocess of this method in the following, which is not described hereinagain.

Certainly, a person skilled in the art may understand that in additionto the foregoing several gaze point detection methods, other methodsthat may be used to detect a gaze point of an eye of the user may alsobe used in the method according to the embodiments of the presentapplication.

Steps of detecting a location of a current gaze point of the user byusing method 3) comprise:

acquiring at least one fundus image at the fundus of an eye of the user;

adjusting at least one imaging parameter of an optical path between animage acquisition location of the at least one fundus image and the eyeof the user until a fundus image that satisfies at least one definedsecond clarity criterion is acquired; and

analyzing the at least one fundus image to obtain the imaging parameterof the optical path and at least one optical parameter of the eyecorresponding to the fundus image that satisfies the at least onedefined second clarity criterion, and calculating, the location of acurrent gaze point of the user relative to the user.

As for the second clarity criterion described herein, the claritycriterion is the foregoing clarity criterion commonly used by a personskilled in the art, which may be the same as or maybe different from theforegoing first clarity criterion.

By analyzing the fundus image, the optical parameters of the eye at thetime when the fundus image that satisfies the at least one definedsecond clarity criterion is acquired is obtained, and therefore, aposition of a current focusing point of a sight line is obtained throughcalculation, which provides a basis for further detecting an observingbehavior of an observer based on the accurate location of the focusingpoint.

The image presented at “the fundus” herein mainly is an image presentedon a retina, which may be an image of the fundus itself, or an image ofanother object projected onto the fundus, such as a light spot patternmentioned below.

The adjusting at least one projection imaging parameter of an opticalpath between the projection location and an eye of the user comprises:adjusting a focal length of at least one optical device on the opticalpath and/or a location of the at least one optical device on the opticalpath, to cause the fundus image that satisfies the at least one definedsecond clarity criterion to be obtained when the at least one opticaldevice is at a certain location or in a certain state, wherein theadjustment may be continuous and in real time.

In a possible implementation manner of an embodiment of the presentapplication, the optional device may be a focal length adjustable lens,configured to adjust a refractive index and/or a shape of the opticaldevice to complete adjustment of the focal length of the optical device.Specifically: 1) Adjust the focal length by adjusting curvature of atleast one surface of the focal length adjustable lens, for example,adjust the curvature of the focal length adjustable lens by increasingor decreasing liquid media in a cavity formed by two transparent layers;2) Adjust the focal length by changing the refractive index of the focallength adjustable lens, for example, a specific liquid crystal medium isfilled in the focal-length adjustable lens, and arrangement of theliquid crystal medium is adjusted by adjusting a voltage of acorresponding electrode of the liquid crystal medium, and therefore,change the refractive index of the focal length adjustable lens.

In another possible implementation manner of the method according to anembodiment of the present application, an optical device may be a groupof lenses, configured to perform adjustment of a focal length of thegroup of lenses by adjusting relative locations between the lenses inthe group of lenses. Alternatively, one or more lenses in the group oflenses are the foregoing focal length adjustable lenses.

In addition to changing the imaging parameters by changingcharacteristics of the optical device as described above, the imagingparameters may also be changed by adjusting the position of the opticaldevice on the optical path.

In addition, in the method according to an embodiment of the presentapplication, the analyzing the at least one fundus image furthercomprises:

analyzing the at least one fundus image to find the fundus image thatsatisfies at least one defined second clarity criterion; and

calculating an optical parameter of the eye according to the fundusimage that satisfies the at least one defined second clarity criterion,and known imaging parameters when the fundus image satisfying the atleast one defined second clarity criterion is obtained.

By adjusting at least one projection imaging parameter of an opticalpath between the projection location and an eye of the user, the fundusimage that satisfies the at least one defined second clarity criterioncan be acquired, however, the fundus images need to be analyzed to findthe fundus image that satisfies the at least one defined second claritycriterion. The optical parameter of the eye can be obtained throughcalculation according to the fundus image that satisfies the at leastone defined second clarity criterion and the known optical pathparameters.

In the method according to an embodiment of the present application, thestep of detecting a location of a current gaze point of the user mayfurther comprise:

projecting a light spot to the fundus.

The projected light spot may have no specific pattern but only be usedto illuminate the fundus. The projected light spot may further comprisea pattern with abundant features. The abundant features of the patternmay facilitate easy detection and improve detection precision. FIG. 4ais a schematic diagram of a light spot pattern P. The pattern may beformed by a light spot pattern generator, such as frosted glass. FIG. 4bshows an image acquired at the fundus when the light spot pattern P isprojected.

In order not to affect normal viewing of the eye, the light spot can bean eye-invisible infrared light spot. In these cases, in order todecrease interference from other spectrums, lights except theeye-invisible light in the projected light spot may be filtered out.

Correspondingly, the method according to the embodiment of the presentapplication may further comprise the following step:

Control luminance of the projected light spot according to a resultobtained through analysis in the foregoing steps. The analysis resultcomprises, for example, features of the fundus images, a contrast of thefeatures of the fundus images, texture features, and the like.

It should be noted that a special case of controlling the luminance ofthe projected light spot is starting or stopping projection. Forexample, when an observer keeps gazing at a point, the projection may bestopped periodically; or when a fundus of the observer is bright enough,the projection may be stopped, and information at the fundus can be usedto detect a distance from a focusing point of a current sight line of aneye to the eye.

In addition, the luminance of the projected light spot may further becontrolled according to the ambient light.

In the method according to an embodiment of the present application, theanalyzing the at least one fundus image further comprises:

calibrating the image at the fundus, to obtain at least one referenceimage corresponding to the image presented at the fundus. Comparativecalculation is performed on the at least one fundus image and thereference image, to obtain the fundus image that satisfies the at leastone defined second clarity criterion. Herein, the fundus image thatsatisfies the at least one defined second clarity criterion may be anobtained image that is the least different from the reference image. Inthe method according to this implementation manner, a difference betweenthe currently obtained image and the reference image may be calculatedby using an existing image processing algorithm, for example, by using aclassic automatic phase difference focusing algorithm.

The optical parameters of the eye may include an optical axis directionof an eye obtained according to features of the eye at the time when thefundus image that satisfies the at least one defined second claritycriterion is acquired. The features of the eye herein may be acquiredfrom the fundus image that satisfies the at least one defined secondclarity criterion, or may be acquired in another manner. A direction ofa sight line at which an eye of the user gazes may be obtained accordingto the optical axis direction of the eye. Specifically, the optical axisdirection of the eye may be obtained according to the features of thefundus at the time when the fundus image that satisfies the at least onedefined second clarity criterion is obtained, and determining theoptical axis direction of the eye by using the features of the fundusmay be more precise.

When a light spot pattern is projected onto the fundus, a size of thelight spot pattern may be bigger than a fundus visible region or may besmaller than the fundus visible region.

When an area of the light spot pattern is smaller than or equal to thefundus visible region, a classic feature points matching algorithm (forexample, the scale invariant feature transform (SIFT) algorithm) may beused to determine the optical axis direction of the eye by detecting alocation of the light spot pattern in the image, relative to the fundus.

When the area of the light spot pattern is bigger than or equal to thefundus visible region, a location of the light spot pattern in theobtained image, relative to an original light spot pattern (obtainedthrough image calibration) may be used to determine the optical axisdirection or to determine a direction of a sight line of the observer.

In another possible implementation manner of the method according to theembodiments of the present application, the optical axis direction ofthe eye may further be obtained according to features of the pupil atthe time when the fundus image that satisfies the at least one definedsecond clarity criterion is obtained. Features of the pupil herein maybe acquired from the fundus image that satisfies the at least onedefined second clarity criterion, or may be acquired in another manner.The obtaining the optical axis direction of the eye by using the featureof the eye pupil belongs to the prior art, and is not described hereinagain.

In addition, the method according to an embodiment of the presentapplication may further comprise: calibrating the optical axis directionof the eye, so as to more precisely determine the foregoing optical axisdirection of the eye.

In the method according to an embodiment of the present application, theknown imaging parameters comprises at least one fixed imaging parameterand at least one real-time imaging parameter, wherein the at least onereal-time imaging parameter is parameter information of the opticaldevice at the time when the fundus image that satisfies the at least onedefined second clarity criterion is acquired, and this parameterinformation may be obtained in a manner of real-time recording at thetime when the fundus image that satisfies the at least one definedsecond clarity criterion is acquired.

After a current optical parameter of the eye is obtained, a location ofa gaze point of the eye may be obtained with reference to a distancefrom a focusing point of the eye to the eye, which is obtained throughcalculation (specific process is to be described in detail incombination with the apparatus part).

In order to make the user related information seen by the user have athree-dimensional display effect and look more real, in a possibleimplementation manner of the embodiment of the present application, theuser related information may be projected onto the fundus of the userthree-dimensionally in step 160.

As described above, in a possible implementation manner, thethree-dimensional display may be using same information. By performingprojection location adjustment in step S160, information with parallaxis seen by two eyes of the user, generating a three-dimensional displayeffect.

In another possible implementation manner, the user related informationcomprises three-dimensional information respectively corresponding tothe two eyes of the user, and in step S160, corresponding user relatedinformation is separately projected onto the two eyes of the user. Thatis: the user related information comprises left eye informationcorresponding to the left eye of the user and right eye informationcorresponding to the right eye of the user, and during projection, theleft eye information is projected onto the left eye of the user and theright eye information is projected onto the right eye of the user, tocause the user related information seen by the user to have a properthree-dimensional display effect, thus bring better user experience. Inaddition, when the user related information input by the user comprisesthree-dimensional space information, the foregoing three-dimensionalprojection causes the user to see the three-dimensional spaceinformation. For example, when a user needs to make a specific handgesture at a specific location in the three-dimensional space tocorrectly input the user related information, the foregoing methodaccording to an embodiment of the present application causes the user tosee three-dimensional user related information, and know the specificlocation and the specific hand gesture according to the user relatedinformation, and further causes the user to make the hand gesture at thespecific location, as prompted by the user related information. At thistime, even if another person sees the gesture action made by the user,the person cannot know the space information, which causes the userrelated information to have a better confidentiality effect.

As shown in FIG. 5, an embodiment of the present application furtherprovides a user information extraction apparatus 500, comprising:

an image acquisition module 510, configured to acquire an imagecomprising at least one digital watermark;

an information acquisition module 520, configured to acquire userrelated information corresponding to a user and comprised in the atleast one digital watermark in the image; and

a projection module 530, configured to project the user relatedinformation to a fundus of the user.

In the embodiment of the present application, user related informationrelated to a user is acquired from a digital watermark of an image andprojected onto a fundus of the user, to cause the user to confidentiallyacquire, without special memorization, the corresponding user relatedinformation in a scenario in which the corresponding user relatedinformation needs to be used, thereby facilitating use of the user, andimproving user experience.

In order to make the user acquire the user related information morenaturally and conveniently, the apparatus according to an embodiment ofthe present application may be, for example, a wearable device that isused near eyes of the user, such as a pair of smart glasses. When a gazepoint of a sight line of the user falls on the image, the image isautomatically acquired by using the image acquisition module 510, andafter the user related information is obtained, the information isprojected onto the fundus of the user.

In the following, embodiments of the present application furtherdescribe the modules of the foregoing apparatus with reference to thefollowing implementation manners:

In an implementation manner of an embodiment of the present application,the image acquisition module 510 may have various forms. For example:

As shown in FIG. 6a , the image acquisition module 510 comprises ashooting submodule 511, configured to shoot the image.

Herein, the shooting submodule 511 may be, for example, a camera of apair of smart glasses, configured to shoot an image seen by the user.

As shown in FIG. 6b , in another implementation manner of an embodimentof the present application, the image acquisition module 510 comprises:

a first communications submodule 512, configured to receive the imagefrom an external device.

In this implementation manner, the image may be acquired by anotherdevice, and then the image is sent to the apparatus according to anembodiment of the present application; or the image is acquired throughinteraction with a device displaying the image (that is, the devicetransfers displayed image information to the apparatus according to anembodiment of the present application).

In an embodiment of the present application, the information acquisitionmodule 520 may also have various forms. For example:

As shown in FIG. 6a , the information acquisition module 520 comprises:an information extraction submodule 521, configured to extract the userrelated information from the image.

In this implementation manner, the information extraction submodule 521may, for example, analyze the digital watermark in the image by using apersonal private key and a public or private watermark extractionmethod, to extract the user related information.

As shown in FIG. 6b , in another implementation manner of an embodimentof the present application, the information acquisition module 520comprises: a second communications submodule 522, configured to:

send the image to an external device, and receive the user relatedinformation in the image from the external device.

In this implementation manner, the image may be sent to the externaldevice, for example, sent to a cloud server or a third party authorityto extract the user related information in the digital watermark in theimage by using the cloud server or the third party authority, and thenthe user related information is sent back to the second communicationssubmodule 522 in an embodiment of the present application.

Herein, functions of the first communications submodule 512 and thesecond communications submodule 522 may be implemented by a samecommunications module.

In a possible implementation manner of an embodiment of the presentapplication, the image corresponds to a graphical user interfacedisplayed by a device. The graphical user interface comprises an inputinterface of the user related information. For details, refer tocorresponding description in the foregoing method embodiments, which isnot described herein again.

In an embodiment of the present application, the user relatedinformation comprises user authentication information corresponding tothe image. For details, refer to corresponding description in theforegoing method embodiments, which is not described herein again.

In an embodiment of the present application, when the apparatus 500 isbeing used, identity confirmation may be performed on the user. Forexample, when the user uses a pair of smart glasses that can implementthe functions of the apparatus according to an embodiment of the presentapplication, the pair of smart glasses first authenticates the user, tocause the pair of smart glasses to know an identity of the user, andsubsequently, when extracting the user related information by using theinformation extraction module 520, the pair of smart glasses onlyacquires the user related information corresponding to the user. Thatis, as long as the user goes through user authentication with the pairof smart glasses of the user once, the user related information can beacquired from devices of the user or public devices by using the pair ofsmart glasses.

As shown in FIG. 6a , in this implementation manner, the projectionmodule 530 comprises:

an information projection submodule 531, configured to project the userrelated information; and

a parameter adjustment submodule 532, configured to adjust at least oneprojection imaging parameter of an optical path between the informationprojection submodule 531 and an eye of the user until the image of theuser related information formed at the fundus of the user satisfies atleast one defined first clarity criterion.

In an implementation manner, the parameter adjustment submodule 532comprises:

at least one adjustable lens device, a focal length of the at least oneadjustable lens device being adjustable and/or a location of the atleast one adjustable lens device on the optical path.

As shown in FIG. 6b , in an implementation manner, the projection module530 comprises:

a curved beam splitting device 533, configured to transfer,corresponding to locations of a pupil when optical axis directions ofthe eye are different, the user related information to the fundus of theuser.

In an implementation manner, the projection module 530 comprises:

a counter-deformation processing submodule 534, configured to perform,on the user related information, counter-deformation processingcorresponding to locations of a pupil at the time when the optical axisdirections of the eye are different, to cause the fundus to receive theuser related information that needs to be presented.

In an implementation manner, the projection module 530 comprises:

an alignment and adjustment submodule 535, configured to align, at thefundus of the user, the projected user related information with an imageseen by the user.

In an implementation manner, the apparatus further comprises:

a location detection module 540, configured to detect a location, of agaze point of the user, relative to the user, wherein

the alignment and adjustment submodule 535 is configured to align, atthe fundus of the user according to the location, of the gaze point ofthe user, relative to the user, the projected user related informationwith the image seen by the user.

For functions of the submodules of the foregoing projection module,refer to the description of corresponding steps in the foregoing methodembodiment, and examples are provided in embodiments shown in FIG. 7a toFIG. 7d , FIG. 8, and FIG. 9 below.

In an embodiment of the present application, the location detectionmodule 540 may have various implementation manners, for example, anapparatus corresponding to the methods 1) to 3) described in the methodembodiment. An embodiment of the present application further describes alocation detection module corresponding to method 3) by usingimplementation manners corresponding to FIG. 7a to FIG. 7d , FIG. 8, andFIG. 9.

As shown in FIG. 7a , in a possible implementation manner of anembodiment of the present application, the location detection module 700comprises:

a fundus image acquisition submodule 710, configured to acquire at leastone fundus image at the fundus of the eye of the user;

an adjustable imaging submodule 720, configured to adjust at least oneimaging parameter of an optical path between the fundus imageacquisition submodule 710 and the eye of the user until a fundus imagethat satisfies at least one defined second clarity criterion isacquired; and

an image processing submodule 730, configured to analyze the at leastone fundus image, to obtain the imaging parameter of the optical pathbetween the fundus image acquisition submodule and the eye of the userand at least one optical parameter of the eye corresponding to thefundus image that satisfies the at least one defined second claritycriterion, and calculating a location of a current gaze point of theuser relative to the user.

The location detection module 700 analyzes fundus images, to obtainoptical parameters of the eye at the time when the fundus imageacquisition submodule obtains the fundus image that satisfies the atleast one defined second clarity criterion, and therefore, the locationof the current gaze point of the eye can be obtained throughcalculation.

The image presented at “the fundus” herein mainly is an image presentedon a retina, which may be an image of the fundus itself, or an image ofanother object projected onto the fundus. The eye herein may be a humaneye, or may be an eye of another animal.

As shown in FIG. 7b , in a possible implementation manner of anembodiment of the present application, the fundus image acquisitionsubmodule 710 is a micro camera, and in another possible implementationmanner of an embodiment of the present application, the fundus imageacquisition submodule 710 may further use a light sensitive imagingdevice directly, such as a CCD, or a CMOS.

In a possible implementation manner of an embodiment of the presentapplication, the adjustable imaging submodule 720 comprises: anadjustable lens device 721, located at an optical path between the eyeand the fundus image acquisition submodule 710, wherein a focal lengthof the adjustable lens device 721 is adjustable and/or a location on theoptical path is adjustable. The adjustable lens device 721 causes asystem equivalent focal length from the eye to the fundus imageacquisition submodule 710 to be adjustable, and adjustment of theadjustable lens device 721 causes the fundus image acquisition submodule710 to obtain, in a certain location or state of the adjustable lensdevice 721, a fundus image that satisfies at least one defined secondclarity criterion at the fundus. In this implementation manner, theadjustable lens device 721 is adjusted continuously in real time duringdetection.

In a possible implementation manner of an embodiment of the presentapplication, the adjustable lens device 721 is a focal length adjustablelens, configured to adjust a refractive index and/or a shape of theadjustable lens device 721, to complete adjustment of the focal lengthof the adjustable lens device 721. Specifically: 1) Adjust the focallength by adjusting curvature of at least one surface of the focallength adjustable lens, for example, adjust the curvature of the focallength adjustable lens by increasing or decreasing liquid media in acavity formed by two transparent layers; 2) Adjust the focal length bychanging the refractive index of the focal length adjustable lens, forexample, a specific liquid crystal medium is filled in the focal-lengthadjustable lens, and arrangement of the liquid crystal medium isadjusted by adjusting a voltage of a corresponding electrode of theliquid crystal medium, and therefore, change the refractive index of thefocal length adjustable lens.

In another possible implementation manner of an embodiment of thepresent application, the adjustable lens device 721 comprises: a groupof lenses, formed by multiple pieces of lenses, and configured to adjustrelative locations between the lenses in the group of lenses to performadjustment of a focal length of the group of lenses. The group of lensesmay also comprise a lens with an imaging parameter, such as a focallength, of the lens being adjustable.

In addition to the foregoing two manners for changing optical pathparameters of the system by adjusting features of the adjustable lensdevice 721, the optical path parameters of the system may also bechanged by adjusting the location of the adjustable lens device 721 onthe optical path.

In a possible implementation manner of an embodiment of the presentapplication, in order not to affect the experience of the user forviewing an observed object, and in order to cause the system to beapplied to the wearable device in a portable manner, the adjustableimaging submodule 720 further comprises: a beam splitting unit 722,configured to form a light transfer path between the eye and theobserved object, and between the eye and the fundus image acquisitionsubmodule 710. This may fold the optical path, decrease a size of thesystem, and affect other visual experience of the user as little aspossible.

In this implementation manner, the beam splitting unit comprises a firstbeam splitting unit, located between the eye and the observed object,and configured to transmit light from the observed object to the eye,and transfer light from the eye to the fundus image acquisitionsubmodule.

The first beam splitting unit may be a beam splitter, a beam splittingoptical waveguide (comprising an optical fiber), or another suitablebeam splitting device.

In a possible implementation manner of an embodiment of the presentapplication, the image processing submodule 730 comprises an opticalpath calibration unit, configured to calibrate the optical path of thesystem, for example, perform alignment and calibration on an opticalaxis of the optical path, to ensure precision of measurement.

In a possible implementation manner of an embodiment of the presentapplication, the image processing submodule 730 comprises:

an image analysis unit 731, configured to analyze the at least oneimage, to find a fundus image that satisfies at least one defined secondclarity criterion; and

a parameter calculation unit 732, configured to calculate opticalparameters of the eye according to the fundus image that satisfies theat least one defined second clarity criterion and known imagingparameters of the system when the fundus image that satisfies the atleast one defined second clarity criterion is obtained.

In this implementation manner, the adjustable imaging submodule 720causes the fundus image acquisition submodule 710 to obtain a fundusimage that satisfies at least one defined second clarity criterion.However, the fundus image that satisfies the at least one defined secondclarity criterion needs to be found by using the image analysis unit731. In this way, the optical parameters of the eye can be obtainedthrough calculation according to the fundus image that satisfies the atleast one defined second clarity criterion and the known imagingparameters of the location detection module 700. Herein, the opticalparameters of the eye may comprise an optical axis direction of the eye.

In a possible implementation manner of an embodiment of the presentapplication, the system further comprises: a projection submodule 740,configured to project a light spot to the fundus. In a possibleimplementation manner, a micro projector may be used to implement thefunction of the projection submodule.

The projected light spot herein may have no specific pattern but only beused to illuminate the fundus.

In an implementation manner of an embodiment of the present application,the projected light spot may comprise a pattern with abundant features.The abundant features of the pattern may be for easy detection andimprove detection precision. FIG. 4a is a schematic diagram of a lightspot pattern P. The pattern may be formed by a light spot patterngenerator, such as frosted glass. FIG. 4b shows an image shot at thefundus when the light spot pattern P is projected.

In order not to affect normal viewing of the eye, the light spot can bean eye-invisible infrared light spot.

At this time, in order to decrease interference from other spectrums:

an eye-invisible light transmission lens may be disposed on an exitsurface of the projection submodule 740; and

an eye-invisible light transmission lens is disposed on an incidentsurface of the fundus image acquisition submodule 710.

In a possible implementation manner of an embodiment of the presentapplication, the image processing submodule 730 further comprises:

a projection control unit 734, configured to control, according to aresult obtained by the image analysis unit 731, luminance of the lightspot projected by the projection submodule 740.

For example, the projection control unit 734 may adjust the luminanceself-adaptively according to a feature of the at least one imageobtained by the fundus image acquisition submodule 710. Herein, thefeatures of the at least one image include a contrast of image features,a texture feature, and the like.

Herein, a special case of controlling the luminance of the light spotprojected by the projection submodule 740 is turning on or off theprojection submodule 740. For example, when a user keeps gazing at apoint, the projection submodule may be turned off periodically; or whena fundus of the user is bright enough, the light source may be turnedoff, and a distance from a gaze point of a current sight line of an eyeto the eye can be detected by only using information at the fundus.

In addition, the luminance of the light spot projected by the projectionsubmodule 740 may further be controlled by the projection control unit734 according to ambient light.

In a possible implementation manner of an embodiment of the presentapplication, the image processing submodule 730 further comprises: animage calibration unit 733, configured to calibrate the image at thefundus, to obtain at least one reference image corresponding to theimage presented at the fundus.

The image analysis unit 731 performs comparative calculation on the atleast one image obtained by the fundus image acquisition submodule 730and the reference image, to obtain the fundus image that satisfies theat least one defined second clarity criterion. Herein, the fundus imagethat satisfies the at least one defined second clarity criterion may bean obtained image that is the least different from the reference image.In this implementation manner, a difference between the currentlyobtained image and the reference image is calculated by using anexisting image processing algorithm, for example, by using a classicautomatic phase difference focusing algorithm.

In a possible implementation manner of an embodiment of the presentapplication, the parameter calculation unit 732 comprises:

an eye-optical-axis-direction determining subunit 7321, configured toobtain an optical axis direction of an eye according to a feature of theeye at the time when the fundus image that satisfies the at least onedefined second clarity criterion is obtained.

The feature of the eye herein may be acquired from the fundus image thatsatisfies the at least one defined second clarity criterion, or may beacquired in another manner. A direction of a sight line at which an eyeof the user gazes may be obtained according to the optical axisdirection of the eye.

In a possible implementation manner of an embodiment of the presentapplication, the eye-optical-axis-direction determining subunit 7321comprises: a first determining subunit, configured to obtain the opticalaxis direction of the eye according to a feature of the fundus at thetime when the fundus image that satisfies the at least one definedsecond clarity criterion is obtained. Compared with obtaining theoptical axis direction of the eye by using features of a pupil and aneyeball, determining the optical axis direction of the eye by using thefeature of the fundus has high precision.

When a light spot pattern is projected onto the fundus, a size of thelight spot pattern may be bigger than a fundus visible region or may besmaller than the fundus visible region.

When an area of the light spot pattern is smaller than or equal to thefundus visible region, a classic feature points matching algorithm (forexample, the SIFT algorithm) may be used to determine the optical axisdirection of the eye by detecting a location of the light spot patternin the image, relative to the fundus.

When the area of the light spot pattern is bigger than or equal to thefundus visible region, a location, of the light spot pattern in theobtained image, relative to an original light spot pattern (obtained byusing the image calibration unit) may be used to determine the opticalaxis direction or to determine a direction of a sight line of the user.

In another possible implementation manner of an embodiment of thepresent application, the eye-optical-axis-direction determining subunit7321 comprises: a second determining subunit, configured to obtain theoptical axis direction of the eye according to a feature of the pupil atthe time when the fundus image that satisfies the at least one definedsecond clarity criterion is obtained. The feature of the pupil hereinmay be acquired from the fundus image that satisfies the at least onedefined second clarity criterion, or may be acquired in another manner.The obtaining the optical axis direction of the eye by using the featureof the eye pupil belongs to the prior art, and is not described hereinagain.

In a possible implementation manner of an embodiment of the presentapplication, the image processing submodule 730 further comprises: aneye-optical-axis-direction calibration unit 735, configured to calibratethe optical axis direction of the eye, so as to more precisely determinethe foregoing optical axis direction.

In this implementation manner, the known imaging parameters of the=system comprises at least one fixed imaging parameter and at least onereal-time imaging parameter, wherein the real-time imaging parameter isparameter information of the adjustable lens device at the time when thefundus image that satisfies the at least one defined second claritycriterion is acquired, and the parameter information may be obtained ina manner of real-time recording at the time when the fundus image thatsatisfies the at least one defined second clarity criterion is acquired.

The distance from the gaze point of the eye to the eye is calculatedbelow, specifically:

FIG. 7c is a schematic diagram of eye imaging, and with reference to alens imaging formula in a classic optical theory, formula (1) may beobtained from FIG. 7c :

$\begin{matrix}{{\frac{1}{d_{o}} + \frac{1}{d_{e}}} = \frac{1}{f_{e}}} & (1)\end{matrix}$

d_(o) and d_(e) are a distance from a currently observed object 7010 ofan eye to an eye equivalent lens 7030 and a distance from a real image7020 on the retina to the eye equivalent lens 7030, respectively, f_(e)is an equivalent focal length of the eye equivalent lens 7030, and X isa direction of the sight line of the eye (which may be obtained by usingthe optical axis direction of the eye).

FIG. 7d is a schematic diagram of a distance from a gaze point of an eyeto the eye that is obtained according to the known image of the locationdetection module 700 and optical parameters of the eye. In FIG. 7d , alight spot 7040 forms a virtual image (not shown in FIG. 7d ) through anadjustable lens device 721. Assuming that a distance from the virtualimage to the lens is (not shown in FIG. 7d ), the following equation setmay be obtained with reference to the formula (1):

$\begin{matrix}\left\{ \begin{matrix}{{\frac{1}{d_{p}} - \frac{1}{x}} = \frac{1}{f_{p}}} \\{{\frac{1}{d_{i} + x} + \frac{1}{d_{e}}} = \frac{1}{f_{e}}}\end{matrix} \right. & (2)\end{matrix}$

d_(p) is an optical equivalent distance from the light spot 7040 to theadjustable lens device 721, d_(i) is an optical equivalent distance fromthe adjustable lens device 721 to the eye equivalent lens 7030, f_(p) isa focal length value of the adjustable lens device 721, and d_(e) is adistance from the eye equivalent lens 7030 to the adjustable lens device721.

As shown in formula (3), a distance d_(o) from the currently observedobject 7010 (the gaze point of the eye) to the eye equivalent lens 7030may be obtained from (1) and (2):

$\begin{matrix}{d_{o} = {d_{i} + \frac{d_{p} \cdot f_{p}}{f_{p} - d_{p}}}} & (3)\end{matrix}$

The optical axis direction of the eye may be obtained due to previousrecords and according to the foregoing distance from the observed object7010 to the eye that is obtained through calculation, and therefore, thelocation of gaze point of the eye can be easily obtained, which providesa basis for subsequent interaction related to the eye.

FIG. 8 is an embodiment of a location detection module 800 applied to apair of glasses G according to a possible implementation manner of anembodiment of the present application, which comprises content recordedin the implementation manner shown in FIG. 7b . Specifically: as can beseen from FIG. 8, in this implementation manner, a module 800 of thisimplementation manner is integrated at the right side of the pair ofglasses G (which is not limited thereto), and comprises:

a micro camera 810, having a same function as the fundus imageacquisition submodule recorded in the implementation manner of FIG. 7b ,and in order not to affect a sight line of a user for normally viewingan object, the micro camera 810 is disposed at the outer right side ofthe pair of glasses G;

a first beam splitter 820, having a same function as the first beamsplitting unit recorded in the implementation manner of FIG. 7b ,disposed with a certain tilt angle at an intersection point of a gazedirection of an eye A and an incident direction of the camera 810, andconfigured to transmit light entering the eye A from an observed objectand reflect light from the eye to the camera 810; and

a focal length adjustable lens 830, having a same function as the focallength adjustable lens recorded in the implementation manner of FIG. 7b, located between the first beam splitter 820 and the camera 810,configured to adjust a focal length value in real time, to cause thecamera 810 to shoot, at a certain focal length value, a fundus imagethat satisfies at least one defined second clarity criterion at afundus.

In this implementation manner, the image processing submodule is notshown in FIG. 8, and functions of the image processing submodule is thesame as that of the image processing submodule shown in FIG. 7 b.

Generally, the fundus is not bright enough, and therefore, it is betterto illuminate the fundus, and in this implementation manner, the fundusis illuminated by using a light source 840. In order not to affect userexperience, the light source 840 herein is an eye-invisible lightsource, preferably, is a near-infrared light source which does notimpose too much effect on the eye A and to which the camera 810 isrelatively sensitive.

In this implementation manner, the light source 840 is located at theouter side of the right side of a spectacle frame, and therefore, asecond beam splitter 850 and the first beam splitter 820 are required tojointly complete transferring, to the fundus, light emitted by the lightsource 840. In this implementation manner, the second beam splitter 850is located in front of an incident surface of the camera 810, andtherefore, light from the fundus to the second splitter 850 also needsto be transmitted.

As can be seen in this implementation manner, in order to improve userexperience and the acquisition clarity of the camera 810, the first beamsplitter 820 may have the characteristics of high reflectivity toinfrared ray and high transmissivity to visible light. For example, aninfrared reflective film may be disposed at one side, of the first beamsplitter 820, facing the eye A, to achieve the characteristics describedabove.

As can be seen from FIG. 8, in this implementation manner, the locationdetection module 800 is located at one side, of a lens of the pair ofglasses G, away from the eye A, and therefore, the lens may beconsidered as a part of the eye A during calculation of an opticalparameter of the eye, and at this time, there is no need to know opticalcharacteristics of the lens.

In another implementation manner of an embodiment of the presentapplication, the location detection module 800 may be located at oneside, of the lens of the pair of glasses G, close to the eye A, and atthis time, an optical characteristic parameter of the lens needs to beobtained in advance, and an influence factor of the lens needs to beconsidered when the distance from the gaze point to an eye of a user iscalculated.

In this embodiment, the light emitted by the light source 840 isreflected by the second beam splitter 850, transmitted by the focallength adjustable lens 830, and reflected by the first beam splitter820, and then passes through the lens of the pair of glasses G to enterthe eye of the user, and finally arrives at the retina of the fundus.The camera 810 shoots an fundus image at the fundus through a pupil ofthe eye A and an optical path formed by using the first beam splitter820, the focal length adjustable lens 830, and the second beam splitter850.

In a possible implementation manner, other parts of the apparatusaccording to an embodiment of the present application are implemented inthe pair of glasses G, and both the location detection module and theprojection module may comprise: a device having a projection function(such as an information projection submodule of the foregoing projectionmodule, and a projection submodule of the location detection module), animaging device with an imaging parameter being adjustable (such as aparameter adjustment submodule of the foregoing projection module, andan adjustable imaging submodule of the location detection module), andthe like, and therefore, in a possible implementation manner of anembodiment of the present application, functions of the locationdetection module and the projection module are implemented by a samedevice.

As shown in FIG. 8, in a possible implementation manner of an embodimentof the present application, in addition to being configured toilluminate the location detection module, the light source 840 mayfurther be configured to assist in projecting the user relatedinformation as a light source of the information projection submodule ofthe projection module. In a possible implementation manner, the lightsource 840 may separately project invisible light to illuminate thelocation detection module, and visible light to assist in projecting theuser related information. In another possible implementation manner, thelight source 840 may further switch between the invisible light and thevisible light in a time division manner. In still another possibleimplementation manner, the location detection module may use the userrelated information to complete the function of illuminating the fundus.

In a possible implementation manner of an embodiment of the presentapplication, in addition to functioning as parameter adjustmentsubmodules of the projection module, the first beam splitter 820, thesecond beam splitter 850, and the focal length adjustable lens 830 mayfurther function as adjustable imaging submodules of the locationdetection module. Herein, in a possible implementation manner, a focallength of the focal length adjustable lens 830 may be adjusted accordingto regions, wherein different regions respectively correspond to thelocation detection module and the projection module, and focal lengthsmay be different. Alternatively, a focal length of the focal lengthadjustable lens 830 is adjusted as a whole. However, a front end of alight sensitive unit (such as CCD) of the micro camera 810 of thelocation detection module is further provided with other opticaldevices, configured to assist in adjustment of the imaging parameters ofthe location detection module. In addition, in another possibleimplementation manner, it may be configured to cause an optical lengthfrom a light emitting surface of the light source 840 (that is, aprojection location of the user related information) to an eye to be thesame as an optical length from the eye to the micro camera 810, thefocal length adjustable lens 830 is adjusted until the micro camera 810receives the clearest image at the fundus, and the user relatedinformation projected by the light source 840 precisely forms a clearimage at the fundus.

As can be seen from the above, in an embodiment of the presentapplication, functions of the location detection module and theprojection module of the user information acquisition apparatus may beimplemented by using a same device, to cause the whole system to have asimple structure and small size, and be carried conveniently.

FIG. 9 is a schematic structural diagram of a location detection module900 of another implementation manner of an embodiment of the presentapplication. As can be seen from FIG. 9, this implementation manner issimilar to the implementation manner shown in FIG. 8, comprising a microcamera 910, a second beam splitter 920, and a focal length adjustablelens 930. Differences lie in that in this implementation manner, aprojection submodule 940 is configured to project a light spot pattern,and the first beam splitter in the implementation manner of FIG. 8 isreplaced with a curved beam splitter 950 as a curved beam splittingdevice.

The curved beam splitter 950 is used herein to transfer, separatelycorresponding to locations of a pupil at the time when optical axisdirections of an eye are different, an image presented at the fundus toa fundus image acquisition submodule. In this way, the camera can shoota mixed and superimposed image formed from various angles of an eyeball.However, only an image of the part at the fundus passing through a pupilcan be clearly formed on the camera, and other parts are out of focusand fail to form an image clearly, and therefore, the imaging of thepart at the fundus are not interfered severely, and the features of thepart at the fundus may still be detected. Therefore, compared with theimplementation manner shown in FIG. 8, this implementation manner canwell obtain an image at the fundus when the eye gazes at differentdirections, to cause the location detection module of thisimplementation manner to be more widely applied with higher detectionprecision.

In a possible implementation manner of an embodiment of the presentapplication, other parts of the user information extraction apparatusaccording to an embodiment of the present application are implemented inthe pair of glasses G. In this implementation manner, the locationdetection module and the projection module may also be multiplexed.Similar to the embodiment shown in FIG. 8, at this time, the projectionsubmodule 940 may project a light spot pattern and the user relatedinformation simultaneously or by switching in a time division manner; orthe location detection module detects the projected user relatedinformation as the light spot pattern. Similar to the embodiment shownin FIG. 8, in a possible implementation manner of an embodiment of thepresent application, in addition to functioning as parameter adjustmentsubmodules of the projection module, the first beam splitter 920, thesecond beam splitter 950, and the focal length adjustable lens 930 mayfurther function as adjustable imaging submodules of the locationdetection module.

At this time, the second beam splitter 950 is further configured toperform optical path transferring, separately corresponding to locationsof a pupil at the time when optical axis directions of an eye aredifferent, between the projection module and the fundus. After passingthrough the curved second beam splitter 950, the user relatedinformation projected by the projection submodule 940 is deformed, andtherefore, in this implementation manner, the projection modulecomprises:

a counter-deformation processing module (not shown in FIG. 9),configured to perform, on the user related information,counter-deformation processing corresponding to the curved beamsplitting device, to cause the fundus to receive the user relatedinformation that needs to be presented.

In an implementation manner, the projection module is configured toproject the user related information to a fundus of the userthree-dimensionally.

The user related information comprises three-dimensional informationseparately corresponding to two eyes of the user, and the projectionmodule separately projects corresponding user related information to thetwo eyes of the user.

As shown in FIG. 10, in a case in which three-dimensional display isrequired, the user information extraction apparatus 1000 needs todispose two sets of projection modules separately corresponding to twoeyes of a user, comprising:

a first projection module corresponding to the left eye of the user; and

a second projection module corresponding to the right eye of the user.

A structure of the second projection module is similar to a structurecombined with the function of the location detection module and recordedin the embodiment of FIG. 10, and is also a structure that can implementboth the function of the location detection module and the function ofthe projection module, wherein the structure comprises a micro camera1021, a second beam splitter 1022, a second focal length adjustable lens1023, and a first beam splitter 1024 (wherein an image processingsubmodule of the location detection module is not shown in FIG. 10) thathave same functions as those in the embodiment shown in FIG. 10. Adifference lies in that in this implementation manner, the projectionsubmodule is a second projection submodule 1025 that can project userrelated information corresponding to the right eye. The projectionsubmodule may also be configured to detect a location of a gaze point ofan eye of the user, and clearly project, to the fundus of the right eye,the user related information corresponding to the right eye.

A structure of the first projection module is similar to a structure ofthe second projection module 1020, but the first projection module doesnot have a micro camera, and is not combined with the function of thelocation detection module. As shown in FIG. 10, the first projectionmodule comprises:

a first projection submodule 1011, configured to project, to the fundusof the left eye, user related information corresponding to the left eye;

a focal length adjustable lens 1013, configured to adjust an imagingparameter between the first projection submodule 1011 and the fundus, tocause corresponding user related information to be presented at thefundus of the left eye clearly, and to cause the user to see the userrelated information presented in the image;

a third beam splitter 1012, configured to perform optical pathtransferring between the first projection submodule 1011 and the firstfocal length adjustable lens 1013; and

a fourth beam splitter 1014, configured to perform optical pathtransferring between the first focal length adjustable lens 1013 and thefundus of the left eye.

This embodiment causes the user related information seen by the user tobe displayed with a proper three-dimensional effect, thereby bringingbetter user experience. In addition, when the user related informationinput by the user comprises three-dimensional space information, theforegoing three-dimensional projection causes the user to see thethree-dimensional space information. For example, when a user needs tomake a specific hand gesture at a specific location in thethree-dimensional space to correctly input the user related information,the foregoing method according to an embodiment of the presentapplication causes the user to see three-dimensional user relatedinformation, and know the specific location and the specific handgesture, and further causes the user to make a hand gesture, at thespecific location, as prompted by the user related information. At thistime, even if another person sees the gesture action made by the user,the person cannot know the space information, which causes the userrelated information to have a better confidentiality effect.

In addition, the present application further provides a computerreadable medium, comprising a computer readable instruction which, whenexecuted, executes the following operation: executing the operation ofsteps S120, S140, and S160 in the method embodiment shown in FIG. 1.

FIG. 11 is a schematic structural diagram of yet another userinformation extraction apparatus 1100 according to an embodiment of thepresent application. A specific embodiment of the present applicationdoes not limit specific implementation of the user informationextraction apparatus 1100. As shown in FIG. 11, the user informationextraction apparatus 1100 may comprise:

a processor 1110, a communications interface 1120, a memory 1130, and acommunications bus 1140, wherein:

the processor 1110, the communications interface 1120, and the memory1130 communicate with each other by using the communications bus 1140.

The communications interface 1120 is configured to communicate with anetwork element, such as a client.

The processor 1110 is configured to execute a program 1132, andspecifically, may execute related steps in the foregoing methodembodiment.

Specifically, the program 1132 may comprise program code, wherein theprogram code comprises computer operating instructions.

The processor 1110 may be a central processing unit (CPU), or anapplication-specific integrated circuit (ASIC), or one or moreintegrated circuits configured to implement the embodiments of thepresent application.

The memory 1130 is configured to store the program 1132. The memory 1130may comprise a high-speed RAM, or may comprise a non-volatile memory,for example, at least one magnetic disk memory. The program 1132 mayspecifically be configured to cause the user information extractionapparatus 1100 to perform the following steps:

acquiring an image comprising at least one digital watermark;

acquiring user related information corresponding to a user and comprisedin the at least one digital watermark in the image; and

projecting the user related information to a fundus of the user.

For specific implementation of the steps in the program 1132, referencemay be made to corresponding description of corresponding steps andunits in the foregoing embodiment, which is not described herein again.A person skilled in the art may clearly understand that for convenienceand ease of description, for a specific working process of the devicesand modules described above, reference may be made to correspondingprocess description in the preceding method embodiment, which is notdescribed herein again.

As shown in FIG. 12, an embodiment of the present application provides auser information embedding method, comprising:

S1200: Embed at least one digital watermark in an image, wherein thedigital watermark comprises at least one piece of user relatedinformation corresponding to at least one user.

Herein, the digital watermark may be classified into two types, namely,a symmetrical watermark and an asymmetrical watermark, according tosymmetry. Conventionally, a symmetrical watermark is embedded anddetected by using a same key. In this case, once a method and a key fordetection are disclosed, the watermark is removed from a digital carriervery easily. However, an asymmetrical watermark technology embeds awatermark by using a private key, and extracts and verifies thewatermark by using a public key. In this way, an attacker can hardlydestroy or remove, by using the public key, the watermark that isembedded by using the private key. Therefore, in an embodiment of thepresent application, the asymmetrical digital watermark is used.

In a possible implementation manner of an embodiment of the presentapplication, the user related information comprises user authenticationinformation corresponding to the image.

In a possible implementation manner of an embodiment of the presentapplication, the image corresponds to a graphical user interfacedisplayed by a device, wherein the digital watermark comprises userrelated information corresponding to the graphical user interface.

In a possible implementation manner of an embodiment of the presentapplication, the graphical user interface comprises an input interfaceof user related information.

In a possible implementation manner of an embodiment of the presentapplication, the graphical user interface is a lock screen interface ofthe device.

For implementation of the foregoing steps, refer to correspondingdescription in the method embodiments shown in FIG. 1, FIG. 2a to FIG.2c , FIG. 3a , and FIG. 3b , which is not described herein again.

It should be understood that in the embodiments of the presentapplication, the sequence numbers of the foregoing steps do not imply anexecution sequence, and the execution sequence of the steps should bedetermined according to the functions and inherent logic, which is notintended to limit the implementation process of the embodiments of thepresent application.

As shown in FIG. 13, an embodiment of the present application provides auser information embedding apparatus 1300, comprising:

a watermark embedding module 1310, configured to embed at least onedigital watermark in an image, wherein the digital watermark comprisesat least one piece of user related information corresponding to at leastone user.

In a possible implementation manner of an embodiment of the presentapplication, the user information embedding apparatus 1300 is anelectronic device terminal, such as a mobile phone, a tablet computer, anotebook, a desktop, or another user terminal.

The apparatus 1300 further comprises:

a display module 1320, configured to display a graphical user interface,wherein the image corresponds to the graphical user interface (as shownin FIG. 2a and FIG. 3a ); and

the digital watermark comprises at least one piece of user relatedinformation corresponding to the graphical user interface.

In a possible implementation manner of an embodiment of the presentapplication, the graphical user interface comprises an input interfaceof at least one piece of user related information.

For implementation of the functions of the modules of the foregoing userinformation embedding apparatus 1300, refer to corresponding descriptionin the embodiment described above, which is not described herein again.

A person of ordinary skill in the art may be aware that the variousexemplary units and steps of the method described with reference to theembodiments disclosed herein can be implemented by electronic hardware,or a combination of computer software and electronic hardware. Whetherthe functions are executed by hardware or software depends on particularapplications and design constraint conditions of the technicalsolutions. A person skilled in the art can use different methods toimplement the described functions for each particular application, butit should not be considered that the implementation goes beyond thescope of the present application.

When implemented in the form of a software function unit and sold orused as a stand-alone product, the functions may be stored in acomputer-readable storage medium. Based on such understanding, theessence of the technical solutions of the present application, or a partcontributing to the prior art, or a part of the technical solution maybe embodied in the form of a software product. The computer softwareproduct may be stored in a storage medium, and comprises severalinstructions for instructing a computer device (which may be a personalcomputer, a server, or a network device) to execute all or a part of thesteps of the method in any embodiment of the present application. Thestorage medium may be any medium that is capable of storing programcode, such as a USB flash drive, a removable hard disk, a read-onlymemory (ROM), a random access memory (RAM), a magnetic disk, or anoptical disc.

The foregoing implementation manners are merely used to describe thepresent application, and are not intended to limit the presentapplication. A person of ordinary skill in related technical field canmake various changes and variations without departing from the spiritand scope of the present application. Therefore, all equivalenttechnical solutions fall within the scope of the present application,and the patent protection scope of the present application shall besubject to the claims.

What is claimed is:
 1. A user information extraction method, comprising:acquiring an image comprising at least one digital watermark; acquiringuser related information corresponding to a user and comprised in the atleast one digital watermark in the image; projecting the user relatedinformation to a fundus of the user; and adjusting at least oneprojection imaging parameter of an optical path between a projectionlocation of the user related information and an eye of the user until animage of the user related information formed at the fundus of the usersatisfies at least one defined first clarity criterion.
 2. The methodaccording to claim 1, wherein the user related information comprisesuser authentication information corresponding to the image.
 3. Themethod according to claim 1, wherein the image corresponds to agraphical user interface displayed by a device.
 4. The method accordingto claim 3, wherein the graphical user interface comprises an inputinterface of the user related information.
 5. The method according toclaim 1, wherein the acquiring an image comprising at least one digitalwatermark comprises: acquiring the image by means of shooting.
 6. Themethod according to claim 1, wherein the acquiring an image comprisingat least one digital watermark comprises: acquiring the image by meansof receiving from an external device.
 7. The method according to claim1, wherein the acquiring user related information corresponding to auser and comprised in the at least one digital watermark in the imagecomprises: extracting the user related information from the image. 8.The method according to claim 1, wherein the acquiring user relatedinformation corresponding to a user and comprised in the at least onedigital watermark in the image comprises: sending the image to anexternal device; and receiving the user related information in the imagefrom the external device.
 9. The method according to claim 1, whereinthe adjusting at least one projection imaging parameter of an opticalpath between a projection location and an eye of the user comprises:adjusting at least one imaging parameter of at least one optical deviceon the optical path and/or a location of the at least one optical deviceon the optical path.
 10. The method according to claim 1, wherein theprojecting the user related information to a fundus of the user furthercomprises: transferring, corresponding to locations of a pupil whenoptical axis directions of the eye are different, the user relatedinformation to the fundus of the user.
 11. The method according to claim10, wherein the projecting the user related information to a fundus ofthe user comprises: performing, on the user related information,counter-deformation processing corresponding to locations of a pupil atthe time when the optical axis directions of the eye are different, tocause the fundus to receive the user related information that needs tobe presented.
 12. The method according to claim 1, wherein theprojecting the user related information to a fundus of the user furthercomprises: aligning, at the fundus of the user, the projected userrelated information with an image seen by the user.
 13. The methodaccording to claim 12, wherein the method further comprises: detecting alocation, of a gaze point of the user, relative to the user, wherein thealigning, at the fundus of the user, the projected user relatedinformation with an image seen by the user comprises: aligning, at thefundus of the user according to the location of the gaze point of theuser, relative to the user, the projected user related information withthe image seen by the user.
 14. The method according to claim 13,wherein the detecting a location, of a gaze point of the user, relativeto the user comprises: acquiring at least one fundus image at the fundusof an eye of the user; adjusting at least one imaging parameter of anoptical path between an image acquisition location of the at least onefundus image and the eye of the user until a fundus image that satisfiesat least one defined second clarity criterion is acquired; and analyzingthe at least one fundus image, to obtain the imaging parameter of theoptical path and at least one optical parameter of the eye correspondingto the fundus image, and calculating a location, of a current gaze pointof the user, relative to the user according to the imaging parametersand the optical parameters of the eye.
 15. The method according to claim14, wherein the adjusting at least one projection imaging parameter ofan optical path between a projection location and an eye of the usercomprises: adjusting a focal length of at least one optical device onthe optical path and/or a location of the at least one optical device onthe optical path.
 16. The method according to claim 14, wherein thedetecting a location, of a gaze point of the user, relative to the userfurther comprises: transferring, corresponding to locations of a pupilwhen optical axis directions of the eye are different, the image at thefundus of the user to the image acquisition location at the fundus. 17.The method according to claim 14, wherein the detecting a location, of agaze point of the user, relative to the user further comprises:projecting a light spot pattern to the fundus.
 18. The method accordingto claim 1, wherein the projecting the user related information to afundus of the user comprises: projecting the user related information tothe fundus of the user three-dimensionally.
 19. The method according toclaim 18, wherein the user related information comprisesthree-dimensional information separately corresponding to two eyes ofthe user; and the projecting the user related information to a fundus ofthe user comprises: separately projecting corresponding user relatedinformation to the two eyes of the user.
 20. A user informationextraction apparatus, comprising: an image acquisition module,configured to acquire an image comprising at least one digitalwatermark; an information acquisition module, configured to acquire userrelated information corresponding to a user and comprised in the atleast one digital watermark in the image; and a projection module,configured to project the user related information to a fundus of theuser, wherein the projection module comprises: an information projectionsubmodule, configured to project the user related information; and aparameter adjustment submodule, configured to adjust at least oneprojection imaging parameter of an optical path between the informationprojection submodule and an eye of the user until an image of the userrelated information formed at the fundus of the user satisfies at leastone defined first clarity criterion.
 21. The apparatus according toclaim 20, wherein the image corresponds to a graphical user interfacedisplayed by a device.
 22. The apparatus according to claim 21, whereinthe graphical user interface comprises an input interface of the userrelated information.
 23. The apparatus according to claim 20, whereinthe image acquisition module comprises: a shooting submodule, configuredto shoot the image.
 24. The apparatus according to claim 20, wherein theimage acquisition module comprises: a first communications submodule,configured to receive the image from an external device.
 25. Theapparatus according to claim 20, wherein the information acquisitionmodule comprises: an information extraction submodule, configured toextract the user related information from the image.
 26. The apparatusaccording to claim 20, wherein the information acquisition modulecomprises: a second communications submodule, configured to: send theimage to an external device; and receive the user related information inthe image from the external device.
 27. The apparatus according to claim20, wherein the parameter adjustment submodule comprises: at least oneadjustable lens device, a focal length of the at least one adjustablelens device being adjustable and/or a location of the at least oneadjustable lens device on the optical path being adjustable.
 28. Theapparatus according to claim 20, wherein the projection module furthercomprises: a curved beam splitting device, configured to transfer,corresponding to locations of a pupil when optical axis directions ofthe eye are different, the user related information to the fundus of theuser.
 29. The apparatus according to claim 28, wherein the projectionmodule further comprises: a counter-deformation processing submodule,configured to perform, on the user related information,counter-deformation processing corresponding to locations of a pupil atthe time when the optical axis directions of the eye are different. 30.The apparatus according to claim 20, wherein the projection modulefurther comprises: an alignment and adjustment submodule, configured toalign, at the fundus of the user, the projected user related informationwith an image seen by the user.
 31. The apparatus according to claim 30,wherein the apparatus further comprises: a location detection module,configured to detect a location, of a gaze point of the user, relativeto the user, wherein the alignment and adjustment submodule isconfigured to align, at the fundus of the user according to thelocation, of the gaze point of the user, relative to the user, theprojected user related information with the image seen by the user. 32.The apparatus according to claim 31, wherein the location detectionmodule comprises: a fundus image acquisition submodule, configured toacquire at least one fundus image at the fundus of an eye of the user;an adjustable imaging submodule, configured to adjust at least oneimaging parameter of an optical path between the fundus imageacquisition submodule and the eye of the user until a fundus image thatsatisfies at least one defined second clarity criterion is acquired; andan image processing submodule, configured to analyze the at least onefundus image, to obtain the imaging parameter of the optical path and atleast one optical parameter of the eye corresponding to the fundusimage, and calculating a location, of a current gaze point of the user,relative to the user according to the imaging parameters and the opticalparameters of the eye.
 33. The apparatus according to claim 32, whereinthe adjustable imaging submodule comprises: an adjustable lens device, afocal length of the adjustable lens device being adjustable and/or alocation of the adjustable lens device on the optical path between theimage acquisition location at the fundus and the eye being adjustable.34. The apparatus according to claim 32, wherein the adjustable imagingsubmodule comprises: a curved beam splitting device, configured totransfer, corresponding to locations of a pupil when optical axisdirections of the eye are different, the image at the fundus of the userto the image acquisition location at the fundus.
 35. The apparatusaccording to claim 32, wherein the location detection module furthercomprises: a projection submodule, configured to project a light spotpattern to the fundus.
 36. The apparatus according to claim 32, whereina function of the location detection module and a function of theprojection module are implemented by a same device.
 37. The apparatusaccording to claim 20, wherein the projection module is configured to:project the user related information to the fundus of the userthree-dimensionally.
 38. The apparatus according to claim 37, whereinthe user related information comprises three-dimensional informationseparately corresponding to two eyes of the user; and the projectionmodule is configured to separately project corresponding user relatedinformation to the two eyes of the user.
 39. The apparatus according toclaim 20, wherein the apparatus is a pair of glasses.
 40. Anon-transitory computer readable storage medium, wherein the computerreadable storage medium comprises an executable instruction, and when acentral processing unit (CPU) of a wearable device executes theexecutable instruction, the executable instruction is used to cause thewearable device to perform the following method: acquiring an imagecomprising at least one digital watermark; acquiring user relatedinformation corresponding to a user and comprised in the at least onedigital watermark in the image; and projecting the user relatedinformation to a fundus of the user; and adjusting at least oneprojection imaging parameter of an optical path between a projectionlocation of the user related information and an eye of the user until animage of the user related information formed at the fundus of the usersatisfies at least one defined first clarity criterion.
 41. A userinformation extraction apparatus, comprising a CPU and a memory, whereinthe memory stores a computer executing instruction, the CPU is connectedto the memory by using a communications bus, and when the userinformation extraction apparatus runs, the CPU executes the computerexecuting instruction stored in the memory, to cause the userinformation extraction apparatus to perform the following method:acquiring an image comprising at least one digital watermark; acquiringuser related information corresponding to a user and comprised in the atleast one digital watermark in the image; projecting the user relatedinformation to a fundus of the user; and adjusting at least oneprojection imaging parameter of an optical path between a projectionlocation of the user related information and an eye of the user until animage of the user related information formed at the fundus of the usersatisfies at least one defined first clarity criterion.